Image combining system, image combining method, and program

ABSTRACT

A data processing device calculates an image conversion parameter corresponding to each partial image. According to the image conversion parameter calculated, a partial image which has been imaged according to the current partial image is geometrically converted. The images are combined to generate a mosaic image. Moreover, the current partial image and at least a part of the mosaic image generated are displayed on a display unit.

TECHNICAL FIELD

The present invention relates to an image combining system, an imagecombining method, and an image combination program for dividing asubject into a plurality of partial images, photographing the dividedpartial images, and combining these partial images to generate an imagehaving a wider viewing angle and a higher definition.

BACKGROUND ART

Conventionally, image combining approaches have been known for dividinga subject into a plurality of partial images which are photographed, andcombining the partial images with one another with estimation of apositional relationship among the respective partial images to generatean image called a mosaic image which has a wider viewing angle and ahigher definition.

For example, Non-Patent Document 1 (Naoki Chiba, Hiroshi Kayano,Mitihiko Mino, Shouji Yasuda, “Image Mosaicking Based on ImageFeatures,” Transaction of The Institute of Electronics, Information andCommunication Engineers D-II, Vol. J82-D-II, No. 10, pp.1581-1589)discloses a method of generating a mosaic image by estimating apositional relationship among respective partial images based on motionvectors within the partial images generated by an image analysis.Specifically, sites (corresponding points) which are similar indistribution of luminance value are found from two partial images whichwere photographed at nearest times, and image conversion parameters arecalculated for making a geometrical conversion (movement, rotation,scaling up/down) such that all of them overlap each other. Then, onepartial image is used as a basis to geometrically convert the otherpartial image using the calculated image conversion parameters togenerate a mosaic image.

The following model, which supposes that a subject is a flat surface, isoften used for the image conversion parameters because of itssimplicity. $\begin{matrix}{\begin{pmatrix}u_{2} \\v_{2} \\1\end{pmatrix} = {\begin{pmatrix}a & b & c \\d & e & f \\g & h & 1\end{pmatrix}\begin{pmatrix}u_{1} \\v_{1} \\1\end{pmatrix}}} & \left\lbrack {{Equation}\quad 1} \right\rbrack\end{matrix}$where (u1, v1, 1), (u2, v2, 1) are homogeneous coordinates of respectivecorresponding points, a, b, c, d, e, f, g, h are image conversionparameters, and t is an arbitrary constant.

Also, for further simplifying the model, a restraint condition may beapplied to the image conversion parameters, such as g=h=0, a=e=cosθ,-b=d=sin θ, and the like.

In the approach described in the aforementioned Non-Patent Document 1, acamera must be moved such that an overlapping portion exists between twopartial images. Otherwise, no corresponding points would exist betweenthe partial images, resulting in a failure in calculating the imageconversion parameters. Also, if part of the subject is not photographedwhen the partial images are photographed, a mosaic image lacks thepartial images, so that a user must exhaustively photograph the subject.

Patent Document 1 (Japanese Patent No. 2744265 (pages 5-7, and FIG. 3)describes an example of an image combining system for displaying ascreen to guide a user having a camera to take a photograph such thatthe foregoing condition will be satisfied. The configuration of thisconventional image combining system described in Patent Document 1 isillustrated in FIG. 1.

In FIG. 1, 101 designates a camera body; 102 a shutter button; 103 adisplay device, 104 and 105 a switch for instructing whether the camerais panned in the left-hand direction or right-hand direction; and 106 alens for focusing a subject on a film or an imager device.

FIG. 2 illustrates an exemplary display by display device 103 of theimage combining system illustrated in FIG. 1.

For example, when the camera is panned in the right-hand direction, asthe right-hand direction is entered through switch 105, right side area202 of partial image 201, which has been already photographed, isdisplayed in a left end area of displayed image 203. Further, left sidearea 205 of partial image 204 currently captured by the camera iscombined in the remaining right side screen of displayed image 203 as amoving image, and the resulting image is displayed.

The user moves (pans) camera body 101 in parallel with an imaging plane,with reference to displayed image 203 on display device 103, such thatright side area 202 of previously photographed partial image 201 is notdiscontinuous to left side area 205 of partial image 204 currentlycaptured by the camera at the boundary therebetween, and depressesshutter button 102 to photograph the partial image. In this way, themosaic image is generated by moving the camera toward a site of thesubject not yet photographed, while including an overlapping areabetween the previously photographed partial image and the partial imagecurrently captured by the camera, and sequentially photographing partialimages.

However, in the conventional image combining system described in theabove-mentioned Patent Document 1, the camera body must be moved in thevertical direction and horizontal direction, for example, if an imagecombining system is manufactured for photographing a subject which isbroad in both the vertical and horizontal directions, based on thetechnique of Patent Document 1. Since a direction in which the camera ismoved must be previously specified by the switch, a problem arises inthat the operation is very complicated. Accordingly, the image combiningsystem described in Patent Document 1 is of a configuration which is notsuitable for photographing a subject which is broad in the vertical andhorizontal directions.

Also, in the image combining system described in Patent Document 1, thecamera body must be moved in parallel with the imaging plane, and theamount of movement must be adjusted such that a previously photographedpartial image is not discontinuous to a partial image currently capturedby the camera at the boundary therebetween. Thus, the user is requiredto perform subtle operations, causing a problem that the user experienceconsiderable difficulty taking photographs.

DISCLOSURE OF THE INVENTION

It is therefore an object of the present invention to provide an imagecombining system and method which are capable of generating a mosaicimage through simple operations without the need for complicated andsubtle operations such as manipulations of a switch, parallel movementsof a camera, alignment between partial images, and the like by the user.

To achieve the above object, in the present invention, a data processingunit calculates image conversion parameters corresponding to eachpartial image, geometrically converts photographed partial images on thebasis of a current partial image based on the calculated imageconversion parameters, and combines them to generate a mosaic image.Also, a display device displays each current partial image and at leastpart of the generated mosaic image.

In such a configuration, the user is not required to previously specify,with a switch, a direction in which a camera is moved, or to move thecamera in parallel with an imaging plane, or to adjust a camera panningto avoid a discontinuous image at the boundary of a current partialimage with a photographed partial image, as required in a conventionalimage combining system. Consequently, a mosaic image can be generatedeven for a subject which is broad in the vertical and horizontaldirections, through simple manipulation, without the need forcomplicated manipulations or alignment by the user.

Also, in the present invention, each image conversion parameter isoptimized so as to maintain the consistency of the geometric conversionand combination among all partial images. Accordingly, shifts in thecombination are eliminated between partial images which werephotographed at separate times, thus achieving a mosaic image with lessdistortion. Further, the mosaic image can be generated at a higherresolution by applying super resolution processing to each partialimage.

The present invention employs, as a displayed image, an image which hasa current partial image placed over the entire screen of the displaydevice, and a mosaic image generated from photographed partial imagesincluding the current partial image displayed in a portion of the screenat a reduced scale; or an image which has a current partial image placedat the center of the screen of the display device, and photographedpartial images included in a predetermined range centered around thecurrent partial image, which are combined for display; or an image whichhas a current partial image placed over the entire screen of the displaydevice, and a portion overlapping with photographed partial imagesdisplayed and highlighted.

As a result, since not only the current partial image but also at leastpart of the mosaic image are displayed on the display device, the usercan readily confirm an area of the subject which has not beenphotographed, and can readily determine in which direction the camerashould be next moved. Therefore, even when photographing a subject whichis broad in the vertical and horizontal directions, the user will notforget to photograph part of the subject.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a side view illustrating the configuration of a conventionalimage combining system.

FIG. 2 is a schematic diagram illustrating an exemplary display by theimage combining system illustrated in FIG. 1.

FIG. 3 is a block diagram illustrating the configuration of a firstembodiment of an image combining system of the present invention.

FIG. 4 is a flow chart illustrating a processing procedure for the imagecombining system illustrated in FIG. 3.

FIG. 5 is a schematic diagram illustrating a first example of adisplayed image by the image combining system of the present invention.

FIG. 6 is a schematic diagram illustrating a second example of displayedimages by the image combining system of the present invention.

FIG. 7 is a schematic diagram illustrating a third example of displayedimages by the image combining system of the present invention.

FIG. 8 is a block diagram illustrating the configuration of a secondembodiment of the image combining system of the present invention.

FIG. 9 is a flow chart illustrating a processing procedure for the imagecombining system illustrated in FIG. 8.

FIG. 10 is a block diagram illustrating the configuration of a thirdembodiment of the image combining system of the present invention.

FIG. 11 a block diagram illustrating the configuration of a fourthembodiment of the image combining system of the present invention.

FIG. 12 is a block diagram illustrating the configuration of animplementation of the image combining system of the present invention.

BEST MODE FOR CARRYING OUT THE INVENTION

Next, the present invention will be described with reference to thedrawings.

FIRST EMBODIMENT

As illustrated in FIG. 3, an image combining system of a firstembodiment comprises imager device 310 such as a still camera, a videocamera or the like; data processing unit 300 for combining partialimages to generate a mosaic image; and display device 320 such as aliquid crystal panel, a Braun tube or the like for displaying the resultof processing by data processing unit 300. Data processing unit 300 isimplemented by an LSI (Large scale Integrated Circuit), a logic circuitor the like which comprises a calculation function and a memory.

Data processing unit 300 in the first embodiment comprises conversionparameter estimation unit 301, storage unit 302, image combination dataretrieval unit 303, image combination unit 304, and control unit 305.

Conversion parameter estimation unit 301 estimates the geometricconversion required for a combination of a previously photographed(preceding) partial image, among partial images photographed by imagerdevice 310, and a partial image which is currently being photographed,and calculates image conversion parameters therefor. All thephotographed partial images and the image conversion parameterscalculated by conversion parameter estimation unit 301 are respectivelystored in storage unit 302.

Image combination data retrieval unit 303 selects partial imagesrequired for the generation of a display image in accordance with theresolution of display device 320 or a display method (screen layout)specified by the user, and retrieves the selected partial images andimage conversion parameters corresponding thereto from storage unit 302.

Image combination unit 304 geometrically converts the photographedpartial image on the basis of a current partial image using the partialimages and image parameters retrieved by image combination dataretrieval unit 303, and combines them to generate a mosaic image. Also,image combination unit 304 generates image data for displaying an image,which includes the current partial image and at least part of thecombined mosaic image, on display device 320 in accordance with thedisplay method (screen layout) specified by the user.

Control unit 305 controls processing by respective conversion parameterestimation unit 301, storage unit 302, image combination data retrievalunit 303, and image combination unit 304.

Next, a processing procedure for the image combining system of the firstembodiment will be described with reference to FIG. 3 using a flow chartof FIG. 4.

Assume in the following that the user moves imager device 310, a subjectis photographed each time the shutter button is depressed or atpreviously set constant time intervals, and photographed partial imagesare sequentially supplied from imager device 310 to data processing unit300.

As illustrated in FIG. 4, as one partial image of the subjectphotographed by imager device 310 is captured into data processing unit300 (step S401), control unit 305 first accesses storage unit 302 tosearch whether or not there is a partial image which includes the samesubject as that photographed immediately before (step S402). If there isa photographed partial image, control unit 305 estimates the geometricalconversion required for a combination of the current partial image withthe partial image photographed immediately before that, using conversionparameter estimation unit 301, and calculates image conversionparameters required for the geometric conversion (step S403). Then,control unit 305 stores the current partial image and the calculatedimage conversion parameters in association with each other in storageunit 302 (step S404). As a method of calculating the image conversionparameters, for example, the method disclosed in the aforementionedNon-Patent Document 1 may be used.

Alternatively, external parameters of the camera may be used for theimage conversion parameters. In this event, first, in an initial frame,feature points are detected on an image, and three-dimensionalcoordinates are determined for each feature point on the assumption thatthe camera is substantially opposite the subject, and the subject is aflat plane positioned at a certain distance from the camera.

In subsequent frames, external parameters of the camera are calculatedusing, for example, an approach disclosed in Tomokazu Sato, MasayukiKanbara, Naokazu Yokoya, and Haruo Takemura, “Three-Dimensional Modelingof Outdoor Environment from Moving Image Utilizing Multi-Baseline StereoMethod,” Technical Report, PRMU2001-233, February 2002. This approachestimates the external parameters of the camera by trackingpredetermined feature points between a currently captured partial imageand a partial image captured before that.

If there is no photographed partial image in the search processing atstep S402, the flow proceeds to processing at step S408, returning tothe processing at step S401, where the next photographed image iscaptured.

Subsequently, control unit 305 selects a plurality of partial imagesrequired for combination into a mosaic image using image combinationdata retrieval unit 303, retrieves the selected partial images andconversion. parameters corresponding thereto from storage unit 302,respectively (step S405), and transfers them to image combination unit304.

Image combination unit 304 geometrically converts the photographedpartial images on the basis of the current partial image using thepartial images and image conversion parameters retrieved by imagecombination data retrieval unit 303, and combines them to generate amosaic image (step S406). Image combination unit 304 also sends imagedata for displaying the image, including the current partial image andat least part of the generated mosaic image, to display device 320 (stepS407). When the image data is generated for displaying the image, anaverage value or a median value is respectively calculated for eachpixel positioned at the same coordinates, and these values may be usedfor pixel values of display device 320.

Contemplated as an image displayed on display device 320 is for example,a layout which has current partial image 901 placed over the entirety ofdisplay screen 900, and mosaic image 902, which is a combination ofphotographed partial images, superimposed on part of display screen 900,for example, in a lower right area at a reduced scale, as illustrated inFIG. 5.

In this event, for indicating to the user where current partial image901 is located, the circumscribed rectangle of current partial image 903may be highlighted within the mosaic image. Non-photographed area 904which does not have a partial image within mosaic image 902 may beidentified, for example, by setting a particular pixel value (black orthe like). Also, when current partial image 903 is located outside ofmosaic image 902, the entire mosaic image may be scrolled and displayedsuch that partial image 903 fits into mosaic image 902, or mosaic image902 may be displayed at a further reduced scale.

Also, contemplated as a second example of an image displayed on displaydevice 320 is a layout which has the current partial image placed at thecenter of the screen at all times, and photographed partial imageswithin a predetermined range centered therearound which are displayed incombination, as illustrated in FIG. 6.

In this event, data processing unit 300 first places first partial image1001 entered at time n at the center of screen 1000 at that time (FIG.6(a)).

Next, data processing unit 300 places second partial image 1011 enteredat time n+1 at the center of screen 1010 at that time, geometricallyconverts first partial image 1001 entered at time n on the basis ofsecond partial image 1011, and arranges converted partial image 1012 incombination (FIG. 6(b)).

Similarly, data processing unit 300 places third partial image 1021entered at time n+2 at the center of screen 1020 at that time,geometrically converts second partial image 1011 entered at time n+1 onthe basis of third partial image 1021, and arranges converted partialimage 1022 in combination. Further, data processing unit 300geometrically converts first partial image 1001 (or partial image 1012)entered at time n on the basis of third partial image 1021, and arrangesconverted partial image 1023 in combination (FIG. 6(c)).

On the screen, the circumscribed rectangle of partial image 1001, 1011,1021 at each time may be highlighted to indicate the current partialimage to the user. Also, non-photographed area 1002, which does not havea partial image present in each screen, may be identified, for example,by setting it to a particular pixel value (black or the like).

Also, contemplated as a third example of an image displayed on displaydevice 320 is a layout which has a current partial image placed over theentire screen, and a highlighted area which overlaps with photographedpartial images, as illustrated in FIG. 7.

In this event, data processing unit 300 first places first partial image1101 entered at time n over entire screen 1100 at that time (FIG. 7(a)).

Next, data processing unit 300 places second partial image 1111 enteredat time n+1 over entire screen 1110 at that time. Further, dataprocessing unit 300 detects overlapping area 1112 of first partial image1101 entered at time n and second partial image 1111 from imageconversion parameters therebetween, and generates an image havinghighlighted overlapping area 1112 (FIG. 7(b)). A method of highlightingoverlapping area 1112 includes, for example, a method of changing thebrightness of overlapping area 1112, a method of blending a particularcolor in overlapping area 1112, and the like.

Likewise, data processing unit 300 places third partial image 1121entered at time n+2 over entire surface 1120 at that time. Further, dataprocessing unit 300 detects overlapping area 1122 of second partialimage 1111 entered at time n+1 and third partial image 1121 from imageconversion parameters therebetween, and generates an image havinghighlighted overlap area 1122. Further, data processing unit 300calculates an overlapping area of first partial image 1101 entered attime n with third partial image 1121 from image conversion parameterstherebetween, and generates an image with the highlighted overlappingarea (FIG. 7(c)). Here, an area in which a plurality of partial imagesoverlap, such as overlapping area 1123, may be highlighted, for example,by performing such processing as blending with a deeper color inaccordance with the number of partial images therein.

In the third exemplary displayed image illustrated in FIG. 7, a mosaicimage having not only a current partial image but also partial imagesfalling within a predetermined range centered therearound and combinedthereinto, may be displayed in a portion of the screen at a reducedscale, as the displayed image illustrated in FIG. 5.

Which of a variety of the exemplary display images illustrated in FIGS.5-7 should be displayed on display device 320 may be previously set bythe user using a switch or the like, or may be switchable in the middleof photographing.

As image data is sent from image combination unit 304 to display device320, control unit 305 determines whether or not the processing has beencompleted for all partial images required for combination into themosaic image (step S408), and returns to processing at step S401 ifprocessing has not been completed to capture the next partial image, andrepeats the aforementioned processing from step S401 to step S408.Conversely, if processing has been completed for all required partialimages, the mosaic image combination processing is terminated.

Next, description will be given of the effects of the first embodimentof the image combining system of the present invention.

In the image combining system of the first embodiment, the user does notmake alignment between partial images, but data processing unit 300geometrically converts partial images for combination based on imageconversion parameters calculated by conversion parameter estimation unit301, thus making it possible to generate a mosaic image through simplemanipulations, without the need for complicated switch manipulations,parallel movements of the camera, alignment between partial images andthe like by the user.

Also, by displaying images as illustrated in FIGS. 5-7, not only acurrent partial image, but also photographed partial images are alsodisplayed in combination, so that the user can readily confirm anon-photographed area of a subject and can readily determine in whichdirection the camera should be moved. In this way, even whenphotographing a subject which is broad in the vertical and horizontaldirections, the user will not forget to photograph part of the subject.

Further, since required partial images and image conversion parametersare retrieved from storage unit 302 in accordance with the resolution ofdisplay device 320 or a display image specified by the user to generateimage data for the display image, the processing for displaying an imagecan be restrained to a minimally required amount.

SECOND EMBODIMENT

Next, a second embodiment of the image combining system of the presentinvention will be described using the drawings.

As illustrated in FIG. 8, the image combining system of the secondembodiment, like the first embodiment, comprises imager device 510 suchas a still camera, a video camera or the like; data processing unit 500for combining partial images to generate a mosaic image; and displaydevice 520 such as a liquid crystal panel, a Braun tube or the like fordisplaying the result of processing by data processing unit 500. Dataprocessing unit 500 is implemented, for example, by an LSI, a logiccircuit or the like which comprises a calculation function and a memory.

Data processing unit 500 in the second embodiment comprises mosaic imagegeneration unit 501, and overall optimization unit 502.

Mosaic image generation unit 501 comprises functions similar to those ofdata processing unit 300 in the first embodiment, and geometricallyconverts and combines photographed partial images on the basis of acurrent partial image to generate a mosaic image. Overall optimizationunit 502 individually optimizes respective image conversion parametersso as to maintain the consistency for the results of the geometricalconversion and combination among all partial images.

As a method of optimizing image conversion parameters, the followingapproach is used by way of example.

First, for feature points in each partial image, all other partialimages are searched for corresponding points to collect pairs of featurepoints. Here, an i-th pair of feature points is designated{x_(i),x_(i)}.

Subsequently, image conversion parameters between partial images arefound to minimize the evaluation equation shown below, and are definedto be final image parameters. $\begin{matrix}{{\sum\limits_{i}{d^{2}\left( {{\hat{x}}_{i},x_{i}} \right)}} + {d^{2}\left( {{\hat{x}}_{i}^{\prime},x_{i}^{\prime}} \right)}} & \left\lbrack {{Equation}\quad 2} \right\rbrack\end{matrix}$whered({circumflex over (d)} _(i) ,x _(i)),d({circumflex over (x)}′ _(i) ,x′_(i))   (3)shows the Euclid distance:

between point {circumflex over (x)} and point x or between point{circumflex over (x)}′ and point x′  (4)

Also, a point{circumflex over (x)}  (5)indicates the position of point x, the coordinates of which areconverted by the image conversion parameters.

When the image conversion parameters are used as external parameters forthe camera, the image conversion parameters may be found such that theyminimize the evaluation equation shown below, and they may be chosen tobe final image conversion parameters. $\begin{matrix}{\sum\limits_{f}{\sum\limits_{p}{w_{p}{{x_{fp} - {\hat{x}}_{fp}}}}}} & \left\lbrack {{Equation}\quad 6} \right\rbrack\end{matrix}$where f is the number of entered partial images, p is the number oftracked feature points, x_(fp) is the coordinates of feature point p inan f-th frame, and{circumflex over (x)} _(fp)  (7)is the coordinate on the image of the f-th frame onto which featurepoint p is projected. Also, W_(p) is a confidence level of the featurepoint based on a tracking error of the feature point.

For reference, a known Levenberg-Marquardt algorithm (see “NumericalRecipes in C [Japanese version] Numerical Calculation Recipes in Clanguage,” pp. 503-507, Gijutsu-Hyoron Co., Ltd, ISBN4-87408-506-1, andthe like) may be used for the optimization of the image conversionparameters. In this event, values generated by mosaic image generationunit 501 may be used for initial values for the image conversionparameters corresponding to each partial image.

Next, a processing procedure for an image combining system of the secondembodiment will be described with reference to FIG. 8 using a flow chartof FIG. 9.

Assume in the following that the user moves imager device 510, and asubject is photographed each time a shutter button is depressed or atpreviously set constant time intervals, and a photographed partial imageis supplied from imager device 510 to data processing unit 500. Also,assume that the following processing by mosaic image generation unit 501and overall optimization unit 502 is controlled by a control unit, notshown, possessed by mosaic image generation unit 501.

As illustrated in FIG. 9, as one partial image of the subject iscaptured from imager device 510 to data processing unit 500 (step S601),mosaic image generation unit 501 generates a mosaic image in a mannersimilar to the first embodiment, and delivers it to display device 520(step S602).

Upon delivery of image data to display device 502, mosaic imagegeneration unit 501 determines whether or not processing has beencompleted for all partial images required for combination into a mosaicimage (step S603), and returns to processing at step S601, if processinghas not been completed, to repeat processing from step S601 to stepS603.

If processing has been fully completed for required partial images,overall optimization unit 502 is used to optimize each image conversionparameter to eliminate shifts in the combination between partial imageswhich were not photographed at adjacent times (step S604).

Finally, a mosaic image is again generated by geometrically convertingand combining the respective partial images using the image conversionparameters optimized in processing at step S604, and the mosaic imagedata is sent to display device 502 (step S605).

In the foregoing description, the overall optimization processing isexecuted for the image conversion parameters after mosaic imagegeneration unit 501 has terminated geometrical conversion andcombination of all partial images, however, overall optimizationprocessing for the image conversion parameters may be executed each timegeometrical conversion and combination processing have been done for apartial image, or may be executed each time the geometrical conversionand combination processing have been completed for every predeterminednumber of partial images. Alternatively, overall optimization processingfor the image conversion parameters may be executed when a predeterminedcondition is satisfied, such as executed immediately before imagerdevice 510 is largely changed in the panning direction, and the like.

Next, a description will be given of effects of the second embodiment ofthe image combining system of the present invention.

In the image combining system of the first embodiment, image conversionparameters are respectively calculated for partial images which werephotographed at adjacent times, and the partial images are geometricallyconverted in sequence based on these values for combination, so thaterrors of the image conversion parameters may be accumulated to causedistortion in a mosaic image.

In the image conversion system of the second embodiment, since shifts inthe combination are eliminated between partial images which werephotographed at separate times, the resulting mosaic image can be lessdistorted than the first embodiment.

THIRD EMBODIMENT

Next, a third embodiment of the image combining system of the presentinvention will be described using the drawings.

As illustrated in FIG. 10, the image combining system of the thirdembodiment, like the first embodiment, comprises imager device 710 suchas a still camera, a video camera or the like; data processing unit 700for combining partial images to generate a mosaic image; and displaydevice 720 such as a liquid crystal panel, a Braun tube or the like fordisplaying the result of processing by data processing unit 700. Dataprocessing unit 700 is implemented, for example, by an LSI, a logiccircuit or the like which comprises a calculation function and a memory.

Data processing unit 700 of the third embodiment comprises mosaic imagegeneration unit 701, and super resolution image generation unit 702.

Mosaic image generation unit 701, which comprises similar functions asthose of data processing unit 300 in the first embodiment or dataprocessing unit 500 in the second embodiment, geometrically converts andcombines photographed partial images on the basis of a current partialimage to generate a mosaic image. Also, mosaic image generation unit 701individually optimizes respective image conversion parameters so as tomaintain the consistency for the results of the geometrical conversionand combination among all partial images.

Super resolution image generation unit 702 generates a higher resolutionmosaic image than the image combining system of the first embodiment orsecond embodiment by applying super resolution processing tophotographed partial images.

As super resolution processing, the following approach may be employed,as disclosed, for example, in Irani Pelog, “Improving Resolution byImage Registration”, CVGIP: Graphical Models and Image Processing, Vol.53, pp. 231-239, 1991, or the like.

Assume now that there exist m partial images {g_(k)}=g₁, g₂, . . . ,g_(m), and assume that f designates a high-resolution mosaic image whichone wishes to generate. Also, assume that a mosaic image and imageconversion parameters generated by mosaic image generation unit 702 areused for an initial image for mosaic image f, and for respective imageconversion parameters between partial images {g_(k)}.

Super resolution image generation unit 702 first sets initial image f(0)for a mosaic image, and estimates a low-resolution image (g_(k)(0)}corresponding to each partial image {g_(k)} based on the result ofsimulations for the process of geometric conversion and low-resolutionmosaic image generation including blurring data.

Next, super resolution image generation unit 702 calculates differentialimage (g_(k)-g_(k)(0)} between the estimated image and entered image,and adds pixel values of the differential image to correspondinglocations in initial image f(0) to generate updated high-resolutionmosaic image f(1).

This processing is repeated until error function e indicated by thefollowing equation becomes sufficiently small. $\begin{matrix}{{\mathbb{e}}^{(n)} = \sqrt{\sum\limits_{k}{\sum\limits_{({x,y})}\left( {{g\quad{k\left( {x,y} \right)}} - {g_{k}^{(n)}\left( {x,y} \right)}} \right)^{2}}}} & \left\lbrack {{Equation}\quad 8} \right\rbrack\end{matrix}$where n is the number of times the calculation is repeated.

A mosaic image generated by mosaic image generation unit 701 may beenlarged, for example, by a factor of two in the vertical and horizontaldirections, respectively, to create an image for use as the initialimage for the mosaic image, or may be enlarged by a factor equal to orlarger than two for use as the initial image.

Next, a description will be given of effects of the image combiningsystem of the third embodiment of present invention.

The image combining system of the third embodiment can provide a higherresolution mosaic image than the image combining systems of the firstembodiment and second embodiment by comprising super resolution imagegeneration unit 702.

The image combining system of the third embodiment may perform onlymosaic image combination processing which imposes a relatively lightprocessing load (a low calculation cost) while the subject is beingphotographed, and executes super resolution image generation processingand overall optimization processing for image conversion parameterswhich impose a relatively heavy processing load (a high calculationcost) after all partial images have been photographed. In this event, amosaic image update response is prevented from becoming degraded whilethe subject is being photographed.

FOURTH EMBODIMENT

Next, a fourth embodiment of the image combining system of the presentinvention will be described using the drawings.

As illustrated in FIG. 11, the image combining system of the fourthembodiment, like the first embodiment, comprises imager device 810 suchas a still camera, a video camera or the like; data processing unit 800for combining partial images to generate a mosaic image; display device820 such as a liquid crystal panel, a Braun tube or the like fordisplaying the result of processing by data processing unit 800; andrecording medium 830 which records an image combination program forcausing data processing unit 800 to generate a mosaic image.

The image combination program is a program for data processing unit 800to implement processing for each conversion parameter estimation unit301, image combination data retrieval unit 303, image combination unit304, and control unit 305 included in the image combining system of thefirst embodiment. Alternatively, the image combination program is aprogram for data processing unit 800 to implement processing for eachmosaic image generation unit 501, and overall optimization unit 502included in the image combining system of the second embodiment.Alternatively, the image combination program is a program for dataprocessing unit 800 to implement the processing of mosaic imagegeneration unit 701, and super resolution image generation unit 702included in the image combining system of the third embodiment.

Data processing unit 800 of the fourth embodiment is implemented by acomputer which has, for example, a CPU, not shown; a main storage devicefor temporarily holding data required for processing of the CPU; an I/Ounit which is an interface unit with imager device 810, display device820, recording medium 830 and the like; and a communication device whichenables data transmission/reception with a server device or the likethrough a network.

Recording medium 830 may be a magnetic disk, a semiconductor memory, amagnetic tape, a CD (compact disk)-ROM, a DVD (digital versatile disk),or other recording media.

Data processing unit 800 loads the main storage device with the imagecombination program recorded on recording medium 830, and executesprocessing similar to the data processing units described in theaforementioned first to third embodiments by the CPU in accordance withthe image combination program. The image combination program need not benecessarily stored in recording medium 830, but may be stored, forexample, in a server device or the like on a network and downloaded tothe main storage unit from the server device through a network inresponse to a request from data processing unit 800.

Likewise, the image combining system of the fourth embodiment canproduce effects similar to those of the image combining systems shown inthe aforementioned first to third embodiments.

IMPLEMENTATION

Next, an implementation of the image combining system of the presentinvention will be described using the drawings.

This implementation is an example in which the image combining system ofthe third embodiment is applied to a mobile telephone system.

As illustrated in FIG. 12, the mobile telephone system to which theimage combining system of the present invention is applied, comprisesmobile telephone 1230, and server device 1240 connected to mobiletelephone 1230 through a network.

Mobile telephone 1230 comprises CCD camera 1210 which is an imagerdevice capable of shooting a moving image; processor 1200 which is adata processing unit; and liquid crystal panel 1220 which is a displaydevice.

Like the data processing unit of the first embodiment, processor 1200comprises conversion parameter estimation unit 1201, storage unit 1202,image combination data retrieval unit 1203, image combination unit 1204,and control unit 1205.

Server device 1240 in turn comprises the super resolution overalloptimization unit shown in the second embodiment, the super resolutionimage generation unit shown in the third embodiment, and a communicationdevice, not shown, which enables data communications with mobiletelephone 1230 through a network.

Processor 1200 comprised in mobile telephone 1230 comprises a memory,and a CPU for executing processing similar to the data processing unitin the first embodiment, for example, in accordance with the imagecombination program. Server device 1240 in turn comprises a memory, anda CPU for executing processing for each aforementioned overalloptimization unit 1241 and super resolution image generation unit 1242in accordance with a program.

In the mobile telephone system of this implementation, as the user movesmobile telephone 1230 while capturing a subject with CCD camera 1210,partial images of the subject photographed by CCD camera 1210 areapplied to processor 1200 at previously set constant time intervals.

As one partial image of the subject is captured, processor 1200 searchesstorage unit 1202 to see whether or not there is a partial image thatincludes the same subject which was photographed immediately before. Ifno partial image is found, the next photographed partial image iscaptured. Alternatively, if there is a photographed partial image,processor 1200 estimates the geometrical conversion required for acombination of the current partial image with the partial imagephotographed immediately before that in accordance with the approachdisclosed in the aforementioned Non-Patent Document 1, and calculatesimage conversion parameters therefor. Then, the current partial imageand calculated image conversion parameters are accumulated inassociation with each other in storage unit 1202.

Next, processor 1200 selects a plurality of partial images required forcombination into a mosaic image, and retrieves the selected partialimages and image conversion parameters corresponding thereto. In thisimplementation, assume that the second exemplary display imageillustrated in FIG. 6 is displayed on crystal panel 1220. In this event,processor 1200 retrieves partial images located within a predeterminedrange centered around the current partial image and image conversionparameters corresponding thereto from storage unit 1202.

Processor 1200 generates a mosaic image using the retrieved partialimages and image conversion parameters, and displays the secondexemplary display image including the mosaic image on liquid crystalpanel 1220. Then, it is determined whether or not processing has beencompleted for all partial images required for combination into themosaic image, and if processing has not been completed, processor 1200retrieves the next image and repeats the processing described above.Conversely, if processing has been completed for all required partialimages, processor 1200 transmits all the partial images and imageconversion parameters corresponding thereto, as well as a request forprocessing the mosaic image (whether or not the overall optimization andsuper resolution processing are required) to server device 1240 througha network.

Upon receipt of the partial images and image conversion parameters frommobile telephone 1230, server device 1240, in response to a processingrequest from the user, optimizes all the image conversion parameters soas to maintain consistency after combination into the mosaic image byoverall optimization unit 1241 in a manner similar to the secondembodiment, and transmits the optimized image conversion parameters tomobile telephone 1230. In this event, mobile telephone 1230 againcombines a mosaic image using the image conversion parameters receivedfrom the server device, and displays it on display device 1220.

Also, upon receipt of the partial images and image conversion parametersfrom mobile telephone 1230, server device 1240, in response to theprocessing request from the user, executes super resolution processingfor each partial image by super resolution image generation unit 1242 ina manner similar to the third embodiment to generate a high resolutionmosaic image, and transmits the mosaic image to mobile telephone 1230.In this event, mobile telephone 1230 displays the high resolution mosaicimage received from the server device on display device 1220.

According to the mobile telephone system of this implementation, amosaic image having a wide viewing angle and a high definition can begenerated using a low resolution camera equipped in mobile telephone1230.

1. An image combining system comprising: an imager device for dividing asubject into a plurality of partial images, and photographing thepartial images; a conversion parameter estimation unit for estimating ageometric conversion required for a combination between the partialimages, and calculating image conversion parameters for the geometricconversion; a storage unit for storing the partial images and thecalculated image conversion parameters corresponding to the partialimages; an image combination data retrieval unit for selecting partialimages required for generating a mosaic image, and retrieving theselected partial images and image conversion parameters correspondingthereto from said storage unit, respectively; an image combination unitfor geometrically converting photographed partial images on the basis ofa current partial image using the partial images and image conversionparameters retrieved by said image combination data retrieval unit,combining them to generate a mosaic image, and generating image data fordisplaying the current partial image and at least part of the mosaicimage; and a display device for displaying an image in accordance withthe image data generated by said image combination unit.
 2. The imagecombining system according to claim 1, further comprising an overalloptimization unit for optimizing each image conversion parameter so asto maintain the consistency of the geometrical conversion andcombination among all the partial images.
 3. The image combining systemaccording to claim 1, further comprising a super resolution imagegeneration unit for generating an image at a resolution higher than themosaic image.
 4. The image combining system according to claim 1,wherein said image combination unit generates image data for placing thecurrent partial image over the entire screen of said display device, anddisplaying a mosaic image generated from the current partial image andthe photographed partial images in a portion of the screen at a reducedscale.
 5. (canceled)
 6. (canceled)
 7. The image combining systemaccording to claim 4, wherein said image combination unit generatesimage data for scrolling and displaying the mosaic image, or displayingthe same at a reduced scale when the current partial image is positionedoutside the mosaic image, such that the current partial image fits inthe mosaic image.
 8. The image combining system according to claim 1,wherein said image combination unit generates image data for placing thecurrent partial image at the center of the screen of said displaydevice, and displaying a combination of the photographed partial imagesincluded in a predetermined range centered around the current partialimage.
 9. The image combining system according to claim 1, wherein saidimage combination unit generates image data for placing the currentpartial image over the entire screen of said display device, anddisplaying and highlighting a portion overlapping with the photographedpartial images.
 10. The image combining system according to claim 9,wherein said image combination unit generates image data for displayingthe overlapping portion with the photographed partial images in a colorwhich is varied in accordance with the number of overlapping partialimages.
 11. An image combining method for combining a plurality ofpartial images separated from a subject and photographed by an imagerdevice to generate a mosaic image for display on a display device, saidmethod comprising: estimating a geometric conversion required for acombination between the partial images, and calculating image conversionparameters for the geometric conversion; accumulating the partial imagesand the image conversion parameters calculated in correspondence to thepartial images in a storage unit, respectively; selecting partial imagesrequired to generate a mosaic image, and retrieving the selected partialimages and image conversion parameters corresponding thereto from saidstorage unit; geometrically converting photographed partial images onthe basis of the current partial image using the retrieved partialimages and image conversion parameters, and combining them to generate amosaic image; and generating image data for displaying the currentpartial image and at least part of the mosaic image on said displaydevice.
 12. The image combining method according to claim 11, furthercomprising executing overall optimization processing for optimizing eachimage conversion parameter so as to maintain the consistency of thegeometrical conversion and combination among all the partial images. 13.The image combining method according to claim 11, further comprisingexecuting super resolution image generation processing for generating animage at a resolution higher than the mosaic image.
 14. The imagecombining method according to claim 11, comprising: generating imagedata for placing the current partial image over the entire screen ofsaid display device, and displaying a mosaic image generated from thecurrent partial image and the photographed partial images in a portionof the screen at a reduced scale.
 15. (canceled)
 16. (canceled)
 17. Theimage combining method according to claim 14, comprising: generatingimage data for scrolling and displaying the mosaic image, or displayingthe same at a reduced scale when the current partial image is positionedoutside the mosaic image, such that the current partial image fits inthe mosaic image.
 18. The image combining method according to claim 11,comprising: generating image data for placing the current partial imageat the center of the screen of said display device, and for displaying acombination of the photographed partial images included in apredetermined range centered around the current partial image.
 19. Theimage combining method according to claim 11, comprising: generatingimage data for placing the current partial image over the entire screenof said display device, and displaying and highlighting a portionoverlapping with the photographed partial images.
 20. The imagecombining method according to claim 19, comprising: generating imagedata for displaying the overlapping portion with the photographedpartial images in a color which is varied in accordance with the numberof overlapping partial images.
 21. A program for causing a computer toexecute processing for combining a plurality of partial images separatedfrom a subject and photographed by an imager device to generate a mosaicimage for display on a display device, said program causing the computerto execute: processing for estimating a geometric conversion requiredfor a combination of the partial images, and calculating imageconversion parameters for the geometric conversion; processing foraccumulating the partial images and the image conversion parameterscalculated in correspondence to the partial images in a storage unit,respectively; processing for selecting partial images required togenerate a mosaic image, and retrieving the selected partial images andimage conversion parameters corresponding thereto from said storageunit; processing for geometrically converting photographed partialimages on the basis of the current partial image using the retrievedpartial images and image conversion parameters, and combining them togenerate a mosaic image; and processing for generating image data fordisplaying the current partial image and at least part of the mosaicimage on said display device.
 22. The program according to claim 21,further causing the computer to execute overall optimization processingfor optimizing each image conversion parameter so as to maintain theconsistency of the geometrical conversion and combination among all thepartial images.
 23. The program according to claim 21, further causingthe computer to execute super resolution image generation processing forgenerating an image at a resolution higher than the mosaic image. 24.The program according to claim 21, causing the computer to executeprocessing for generating image data for placing the current partialimage over the entire screen of said display device, and displaying amosaic image generated from the current partial image and from thephotographed partial images in a portion of the screen at a reducedscale.
 25. (canceled)
 26. (canceled)
 27. The program according to claim24, causing the computer to execute processing for generating image datafor scrolling and displaying the mosaic image, or displaying the same ata reduced scale when the current partial image is positioned outside themosaic image, such that the current partial image fits in the mosaicimage.
 28. The program according to claim 21, causing the computer toexecute processing for generating image data for placing the currentpartial image at the center of the screen of said display device, anddisplaying a combination of the photographed partial images included ina predetermined range centered around the current partial image.
 29. Theprogram according to claim 21, causing the computer to executeprocessing for generating image data for placing the current partialimage over the entire screen of said display device, and displaying andhighlighting a portion overlapping with the photographed partial images.30. The program according to claim 29, causing the computer to executeprocessing for generating image data for displaying the overlappingportion with the photographed partial images in a color which is variedin accordance with the number of overlapping partial images.